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ABSTRACT 

This paper examines the development of a comprehensive 
assessment system within a school-university educator preparation 
collaborative, detailing the evidence sources used in relation to the three 
categories into which the assessment system is organized (performance, 
process, and program) . The Professional Development School Network (PDSN) was 
created to replace traditional - field experience placements. NCATE PDS draft 
standards framed the PDSN * s development. Four school systems collaborated to 
select 28 participating PDSs. Throughout the program, student teachers worked 
with widely diverse student populations in urban, suburban, and rural PDSs. 
The paper reviews the assessment system in relation to evaluation and inquiry 
and informing and transforming practice. It considers challenges faced by 
accredited educator preparation programs and comprehensive assessment 
systems. It concludes by explaining that through this assessment, data from 
multiple sources affirm and substantiate evaluative judgments regarding 
candidates* performance, program quality, and process effectiveness. The 
assessment system facilitates critical review of the unit's work in relation 
to clearly articulated standards and in light of performance based evidence. 
Four appendixes present the 2002 PDSN profile; impacting student learning 
components, schedule, rubric sample data, spring 2002; apprentice mid-term 
and final evaluation summaries; and PDSN self-study 2000, composite summary 
scores for PDS functions and NCATE draft standards. (Contains 30 references.) 
(SM) 
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With the adoption of the NCATE 2000 Standards, comprehensive systemic 
assessment has become an explicit requirement of nationally accredited educator 
preparation units. As noted in the standard statement: ^^The umt has an assessment 
system that collects and analyzes data on the applicant qualifications, the candidate and 
graduate performance, and unit operations to evaluate and improve the umt and its 
programs.” (NCATE 2000 Unit Standards, pg. 1) The standard clearly identifies the 
interrelated categories to which the assessment system must attend: performance, 
processes, and programs. The elements of these categories are illuminated in a close 
reading of the other standards and the accompanying explanations contained in the 
standards document. 

Regarding performance, the assessment system must collect and analyze evidence 
of pre-admission applicants’ ability, admitted candidates’ knowledge, skills, dispositions, 
and ability to impact P-12 students’ learning, and faculty (both university and school- 
based) performance in relation to program implementation, candidate learning and 
achievement. Process focused assessment must monitor and gauge the effect and 
efficacy of activities and formal procedures on candidate performance, program 
implementation and collaboration among university based teacher educators, arts and 
sciences faculty, and school-based P-12 practitioners including faculty and 
admini strators. Program assessment focuses on content, the enactment of the cumculum, 
field experiences, and their integration, as well as their alignment with the program s 
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explicit standards. Progr am assessment also attends to the defining of tasks that link the 
curriculum and program expectations to candidate performances and the rubrics for 
evaluating those performances. The assessment related language of the NCATE 2000 
Unit Standards document defies the reductionist inclination of current federal policy to 
judge candidate proficiencies and program/unit effectiveness in producing competent 
education professionals by the performance of candidates on a single standardized test. 
Indeed the NCATE standard expects units to develop and employ a system of multiple 
indicators of performance proficiencies, program and process effectiveness. This 
expectation reflects the >vell-substantiated convictions regarding the complex nature of 
teaching, and the absolute necessity of deliberate, coherent integration of the content, 
pedagogy, and application components of professional preparation programs. 

As is evident in the series of papers commissioned by NCATE in relation to 
Standard 2 (see Stiggins 2000, Scannell 2000,Weisenbach 2000, Stroble 2000) the 
processes and considerations inherent in developing an assessment system are complex, 
extensive and intensive, and likely to challenge longstanding institutional and faculty 
traditions. Stiggens, for example, notes that assessment is often a neglected element of 
teacher preparation programs, both in terms of such programs having coherent, 
comprehensive assessment systems in place and with regard to adequately preparing 
candidates for the assessment activities in which teachers regularly engage. He, 
Weisenbach and Stroble all note that the culture of teaching in higher education is 
essentially autonomous and private. A system of assessment -will require considerable 
faculty collaboration, with each other and with school-based colleagues. It requires 
articulating a common vision of expectations, a shared set of achievement standards for 
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candidates, and working simultaneously toward a coherent vision of the program as a 
whole and a shared understanding of the contributions distinct pieces make to enact that 
vision. The entire range of key stakeholders must be engaged in developing and 
implementing the assessment system, utilizing the evidence it generates, and contributing 
to its on-going improvement. Weisenbach recommends building the assessment system 
around the unit’s conceptual framework and clearly articulated standards. These form the 
foundation of the assessment system and the bridges across the performance, process, and 
program categories. She and Stroble both note the potential that building and utilizing 
such assessment systems might hold for cultivating communities of inquiry involving 
candidates as well as university and school-based faculty within the unit. These authors 
further note that in addition to meeting the NCATE 2000 Standards, comprehensive 
assessment systems can provide education units with considerably more substantive, 
valid and reliable evidence that they are effectively preparing candidates for the complex 
responsibilities and challenges of teaching. 

The balance of this paper examines the development of a comprehensive 
assessment system within a school-university educator preparation collaborative; 
detailing the evidence sources used in relation to the three categories — ^performance, 
process, and program — ^into which the assessment system is organized. It then reviews 
the assessment system in relation to evaluation and inquiry, informing and transforming 
practice. It concludes with consideration of challenges faced by accredited educator 
preparation programs and comprehensive assessment systems. 
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Contextual Background 

Like most NCATE accredited institutions, the educator preparation unit described 
in this paper resides in a Masters I (formerly regional comprehensive) public university. 
Teaching is the primary focus of the university and tenure line faculty traditionally carry 
a four-course teaching load per semester. Tenure line faculty in the educator preparation 
programs balance their load between course instruction and field-based responsibilities. 
The educator preparation collaborative encompasses the twelve certification programs 
offered through the Department of Teacher Development within the College of 
Education. It offers undergraduate majors with certification in Early Childhood 
Education (P-5) and Middle Grades Education (4-8). In cooperation with arts and 
sciences departments, it offers secondary certification (7-12 in Biology, Chemistry, 
English, History, Mathematics, Math/Physics, Political Science) and P-12 certification in 
foreign languages and music. Candidates in these programs complete majors in their 
content fields. 

Between 1997and 1998 the department faculty in collaboration with arts and 
sciences colleagues and P-12 faculty and administrators undertook a comprehensive 
review of programs. This review revealed substantial lack of program coherence 
evidenced particularly by near total disconnect across the content, pedagogy, and field 
components of the programs. In Fall 1998 the initial phase of revisions in these 
certification programs was inaugurated. Sunultaneously, a Professional Development 
School Network (PDSN) was collaboratively created to replace the traditional field 
experience placement process. The NCATE PDS Draft Standards (1997) were used to 
fi^e the Network’s development. Four school systems jointly participated in the 
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selection of the 28 PDSs: 17 elementary schools, 6 middle schools, and 5 secondary 
schools from across the four systems. A central feature of the program revisions is that 
all field experiences in all of the certification programs occur in the PDSs. Course related 
field experiences are scheduled into a five-week block in the middle of the fifteen-week 
semester. During this period on-campus class meetings are suspended to facilitate faculty 
participation in the field experiences. Candidates also complete their culminating 
Apprenticeship (student teaching semester) in a PDS with a Network approved master 
teacher. Over the course of their programs candidates work in urban, suburban and rural 
PDSs with widely diverse student populations, (see PDS Network Profile in Appendix A) 
The Conceptual Framework and Assessment System 

“Understanding for Teaching, Teaching for Understanding” is the conceptual 
theme adopted for all educator preparation programs in the unit. The core 
conceptualizations of learning, teaching and assessing for understanding embodied by 
thi s theme were drawn primarily from the work of David Perkins, Vito Perrone, Martha 
Stone Wiske and their colleagues at Harvard University in Project Zero and the Teaching 
for Understanding Project (Wiske 1998). Key propositions and beliefs within this work 
resonate to program values and goals. These included the notions that teachers are the 
primary decision makers about curriculum, that learners must and do construct their own 
understandings, that understanding is the ability to think and act flexibly with what one 
knows, that performances represent the state and quality of understanding, and that 
assessment is an on-going disposition and perspective entailing the identification and 
application of criteria for defining, reviewing and evaluating performances of 
understanding rather than being thought of as a distinct performance, task or event. These 
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core concepts provide a common fiame for the development of each certification 
program’s curriculum, field experiences, and performance requirements. 

A second commonality across the certification programs is the adoption of the ten 
INTASC Standards as the Conceptual Framework Principles (CFP) and primary outcome 
targets. Each program’s curriculum was then developed reflecting appropriate 
distinctions related to eontent, content pedagogy, and developmental pedagogy. Table I 
summarizes the semester hour distribution for each program by these categories. 



Table I 

Distribution of Semester Hours In Educator Preparation Programs 



Program 


General Core 
Arts & Sciences 


Major Specific 
Core 

A&S/Pedagogy 


Upper Division 
Content 


Upper Division 
Content 
Pedagogy 


Upper Division 
Developmental 
Pedagogy 


Early 

Childhood 

P-5 


42 


9/9 


Two 12-15sh 
concentrations 
(Math& 
Reading) 


12 

6 each in 
science and 
social studies 


6 


Middle 

Grades 

4-8 


42 


9/9 


Two 12sh 
concentrations 
(Math, English, 
Social Studies, 
Science) 


9 n 

including 

reading 

diagnosis- 

remediation 


12 


Secondary 

7-12 


42 


9/9 


30 sh single 
subject major 


6 


6 



Within each program, each course identifies its primary objectives in tenns of the 
INTASC/Conceptual Framework Principle(s) most appropriate to its focus. There are no 
generic content pedagogy or developmental pedagogy courses in which students from 
different programs enroll together. Course work and field experiences are completely 
tailored to the content and/or grade level focus of each program. Performance 
requirements within each course are designed for candidates to demonstrate the 
developmental quality of their understandings and proficiencies in relation to the targeted 
principles. Each candidate constructs a portfolio, organized around the ten 
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DSfTASC/Conceptual Framework Principles, that evidences performances and related 
indicators of progress and proficiency. 

Revising the programs and cultivating the collaborative processes and structures 
through which they are enacted have followed an emergent progression over the past four 
years. The same is true of the process through which the assessment system has 
developed. Efforts have been guided by commitments to coherence, authenticity, 
legitimacy and comprehensiveness (Lather 1986, Lieberman 1988, 1992, Little 1988, 
Perrone 1991, Gitlin 1992, Wiggins 1995, Joyce & Showers 1996, Schalock 1998, Schon 
1991). 

Progr am revisions, performance expectations and processes for decision-making 
and program implementation and the assessment system are being enacted within this 
conceptual theme and a framework of standards. In addition to the INTASC and NCATE 
standards this work is informed by consideration of standards, principles and guidelines 
promulgated by national professional societies such as ACEI, NAEYC, NMSA; content 
specific professional societies including NCSS, NCTM, NCTE, NSTA, AAAS; the state 
certification agency’s standards and the university system’s Regents’ Principles and 
Guidelines for Educator Preparation. The NCATE PDS Standards ^both the 1997 Draft 
Standards and the 2001 adopted version — ^figure prominently in the design, 
implementation and assessment of the PDSN as both a process and program component. 

There are currently 30 distinct evidence sources in the assessment system. Each 
is identified in Table II in relation to the category(ies) to which it is linked. Notations 
indicate the frequency of data collection from each source and their relative significance 
in the assessment of a given category. Among the evidence sources there is considerable 
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variability in their activation. The PDS Perceptual Survey, for example, has been in use 
since the first semester, fall 1998, of implementation of the program revisions and PDSN; 
there now exists eight semesters worth of data from this survey. By contrast the 
Employer Satisfaction Survey was first piloted in spring 2002. A more extensive 
explanation of each category of the assessment system and its primary evidence sources 
is provided below. 



Table II 

Comprehensive Ass^sment Framework 



Evidence Sources 


Performance\Frequency 


ProcessVFreq uency 


ProgramVFrequency 


Course Performance 
Assessment 


C 1 


\Semester 




1 


Wmually 


Lab Assessment 


C 1 


\Semester 




1 


\Annually 


Professional Qualities 


C 1 


\Semester 




1 


\Annually 


ISL 


C 1 


\Semester 




1 


\Annually 


Intervention 


C 2 


\As Needed 


1 


\Annually 


1 


\Annually 


Candidate Portfolio 


C 1 


\Semester 




2 


\Annually 


Apprentice Assessment 


C 1 


\Semester 




1 


Wmually 


PRAXIS n Scores 


C 2 


\Annually 




2 


\Annually 


Guarantee Referrals 


C 2 


\Annually 




1 


Wmually 


Course Evaluation 


F 2 


\Semester 






Tenure & Promotion 


F 1 


\As Appropriate 




2 


Wmually 


Master Teacher Evaluation 


F 2 


\Semester 


1 


\Annually 


2 


Wmually 


University Coord. Evaluation 


F 1 


\Semester 


1 


\Annually 


1 


Wmually 


Building Coord. Evaluation 


F 2 


\Semester 


1 


\Annually 


2 


Wmually 


P-12 State Report Card 


S 2 


\Annually 


2 


VAnnually 


2 


Wmually 


PDS Perception Survey 




1 


\Semester 


2 


Wmually 


Candidate Satisfaction Survey 




2 


\Annually 


2 


\Annually 


Graduate Satisfaction Survey 




2 


Wmually 


2 


Wmually 


Employment Rates 






2 


Wmually 


Employer Satisfection Survey 




2 


\Annually 


2 


\Annually 


Master Teacher Selection 




1 


\Annually 


1 


Wmually 


Professional Development 
Participation 




1 


Wmually 


1 


Wmually 


PDSNl Meeting notes 




1 


Wmually 


1 


\Annually 


PDS continuation 




1 


Wmually 


2 


\Annually 


PDS Inquiry Year 




1 


Wmually* 


2 


\Annually* 


PDS Review & Renewal Year 




1 


\Annually* 


2 


Wmually* 


Regents’ Annual Review 




2 


Wmually 


1 


Wmually 


Program Folio Review 






1 


\Every 5 years 


NCATE Accreditation 




1 


\Every 5 years 


2 


\Every 5 years 


Program Enrollment Patterns 




2 


\Biermially 


2 


\Biennially 



Key C=educator preparation camdidate; F=PDSNI faculty; S=P-I2 students; l=primaiy significance; 2-secondaiy significant; 
•=within the overall four year cycle, each PDS does a formal Inquiry in the third year of its cycle and a Review and Renewal Year in 
the fourth year of the cycle. Frequency indicates the timeframe of data collection. 
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Performance Assessments 



The performance category includes evidence sources related to educator 
candidates, university and P-12 faculty who participate in the educator preparation 
programs, and P-12 students who attend the PDSs. Fifteen of the thirty evidence sources 
relate to this category; nin e of these relate to educator preparation candidates. The first 
four — course performance assessment, lab assessment, professional qualities, ISL — 
encompass candidate performance in program course work and integrated field 
experiences (labs). As noted previously the ten INTASC Standards were adopted for all 
educator preparation programs and adapted as primary objectives in courses. Within 
each course performance requirements are keyed to the INTASC Standard they address. 
The faculty has adopted a four point scale reflecting a common evaluation rubric for 
scoring performances. Each course performance assessment summarizes each candidate’s 
performance score for each course objective. At the end of each semester, faculty 
members submit a course performance assessment summary on each candidate to a 
secured electronic data file. The electronic files can be aggregated by course, by 
INTASC/Conceptual Framework Principle, by candidate cohorts, and by program to 
provide various representations of performance assessment. As candidates proceed 
through the program they can review their course performance assessments with their 
advisors to clarify areas needing improvement, more attention, fullerdocumentation, etc. 
Candidates include evidence from specific course performances in their portfolios, which 
are organized by the ten INTASC/Conceptual Framework Principles. These become the 
evidentiary basis for evaluating candidates’ programmatic progress and their proficiency 
on each standard/principle. 




Page 9 

iO 



As part of the revision process that preceded the fall 1998 program 
implementation, university and school-based faculty worked collaboratively to improve 
the integration of field experiences with courses, in part, through the alignment of 
performance tasks and course objectives. Course syllabi revisions reflected a 
commitment to the lab experiences as the opportunity for candidates to critically apply 
classroom knowledge, strategies, and propositions regarding teaching and learning. The 
strategy adopted across courses and programs to convey the sigmficance of lab 
performance expectations was to ensure that students had to perform satisfactorily in the 
lab in order to pass the course. Faculty configured the relative weight of graded 
assignments to reflect this requirement. Through this strategy school-based application 
and related reflection performances were more rigorously structured and gained 
considerable significance with candidates. 

The introduction of the Impacting Student Learning (ISL) component into each 
program in Fall 2001 further structured and focused the lab performance expectations. 
Appendix B summarizes the current versions of the ISL elements, scoring rubric and 
form, and a table showing the distribution of ISL requirements within each program. 
Although candidates include evidence of student learning and achievement in their 
completed ISLs, the evaluation of the candidate’s performance on an ISL is not reduced 
to specific evidence of gains by the P-12 students. In designing the ISL faculty were 
deliberate to avoid that distortive reductionism. They were intent on helping candidates 
link their instructional planning and delivery to evidence of student progress and 
achievement, and then to reflect on these integrated elements of instruction/leaming to 
inform their self-evaluations and subsequent iterations of instructional planning and 
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action. Reflections on the preliminary ISLs by both candidates and faculty indicate that 
they enrich candidates’ comprehension of each element of the complexity of teaching 
(clarifying instructional goals, understanding content as curriculum with sufficient depth 
and facility to select materials, strategies and sequences of activities with the goals and 
knowledge of their students as learners in mind, planning assessments that align with 
these other elements), their interactions and interdependence. The anecdotal evidence 
accumulated fi"om the first two semesters of implementation indicates the ISL shifts 
candidates’ focus fi'om themselves as “students” — meeting requirements and the 
expectations of others— to their students. Candidates are increasingly focused, even 
those in the first semester of the program, on the impact their efforts are having on their 
P-12 students’ learning and achievement — a pivotal performance expectation of the 
program. The ISLs include evidence of P-12 student learning and achievement as the 
sampled data summaries in Appendix B illustrate. Assessment of candidate performance 
and effectiveness has not, however, been reduced to this single measure. 

The “lab” teacher with whom a candidate has worked during the intensive five- 
week field experience completes the Lab Assessment and Professional Qualities 
instruments. The Lab Assessment instrument provides a holistic rating of the candidates’ 
performance in the classroom in relation to the targeted INTASC/Conceptual Framework 
Principles. It also includes indicators of the candidate’s oral and wntten language skills. 
These scores are notated in the course performance assessment data set. The Professional 
Qualities data are maintained in the candidate’s file, primarily for use by the candidate 
and his/her advisor. This data set has not yet been formally compiled for aggregation to 
cohort or program level, although it could be quite readily. 




Page 1 1 

12 



The Apprentice Assessment instrument summarizes the candidate’s and his/her 
master teacher’s evaluation of performances within the intensive culminating semester. 
The instrument’s categories and items are keyed to the INTASC Standards and the same 
four-point scale for scoring performances that is used in course work is used here. 
Candidates and their master teachers each complete the form at mid-term and semester’s 
end. The completed instrument reflects an Apprentice portfolio in which the candidate 
has included performance evidence of progress and proficiency. Appendix C includes 
summary data from a random sample of Apprentices’ mid-term and final evaluation 
scores. 

Interventions are formal procedures undertaken to encourage and support 
candidates identified with significant professional qualities or course/lab performance 
difficulties. The intervention reflects counseling with the candidate, a plan and timeline 
for addressing the difficulties, and indicators for determining successful resolution. 
Interventions are maintained in the candidate’s file. At the end of each semester the 
department chair and faculty summarize the status of each intervention. The data set 
indicates any patterns of difficulties, origins (class or lab), numbers of interventions and 
successful resolutions by candidate, cohort, and program. With regard to the 
performance assessment category this data set reveals candidate’s relative success with 
owning, acting on and resolving a difficulty that is perceived by faculty to warrant formal 
deliberate attention. Candidates are counseled that what they do to address and resolve 
an intervention is the most critical issue, not that an intervention has been initiated. 

In Spring 2001 the educator preparation programs discussed in this paper began 
implementation of the university system’s requirement of the “graduate guarantee”. The 
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performance expectations and rubric in the guarantee mirror the ESfTASC/Conceptual 
Framework Principles; a set of procedures for referral by principals also is spelled out in 
the guarantee document. There have been no referrals to date, which is, itself, one form 
of data. Should referrals be made the procedure will allow for collecting data on areas of 
deficiency that can be linked to program elements and to the individual’s performance 
record while in the program. 

It should be noted that faculty have not relinquished fundamental course and 
instructional prerogatives. Individual faculty continue to organize course content, set 
objectives, determine materials to be used, and plan performance tasks that will figure 
into the end of course summative evaluation of candidates. Each instructor continues to 
determine the weight each such task will carry into the final calculation of a course grade 
and proficiency scores. Review of syllabi reveals a wide range of performances 
including traditional tests, essays, research projects, as well as artifacts of varied 
application tasks. The adoption of the common elements— weighting the lab experience 
requirements, the ISL outline, rubric, and inclusion pattern, the adoption of a common 
scoring scale, the utilization of the course performance assessment for data compilation 
reflects a collegial effort to clarify and connect curriculum components of programs. 
These commonalities facilitate communication and comprehension among all 
participants: P-12 and university-based faculty, and candidates. They are a consequence 
of deliberate faculty collaboration in pursuit of more transparent program coherence. 
This work reflects the type of transformation Stiggens (2000) suggests will attend to 
development of a fully integrated assessment system. It has altered relationships 
between P-12 and university faculty but not in the detrimental way projected in some of 
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the literature on school — ^university partnerships (Stoddard 1993, Bulloch, et al. 1997, 

Tom 1998). 

The PRAXIS II exam is required not by the program but by the state for 
certification. One difficulty with using PRAXIS II scores as a primary performance 
measure is that in all but one program the required exams focus exclusively on content 
preparation. The state does not require completion of the pedagogy/leaming tests in the 
PRAXIS n series for certification, so those components of the educator preparation 
programs are not considered in this high-stakes certification requirement. The most 
compelling limitation of using PRAXIS II scores as a performance measure is that it does 
not authentically articulate to instructional practice. Nevertheless review of cohort 
performance and subtest scoring patterns on PRAXIS II can contribute to program review 
particularly with regard to alignments between the exam and pertinent components of the 
curriculum — in terms of both what each addresses and how it does so. 

The evidence sources included for umversity-based faculty assessments include 
the annual departmental evaluation, which is based on individual faculty reports of 
progress and achievement in the areas of teaching, service, and professional development. 
Evidence includes exemplars of scholarship, course and curriculum development, and 
course evaluations completed for every course taught during the academic year. It also 
includes documentation of work in the PDSN. These procedures comply with umversity- 
wide faculty performance evaluation policies. The umversity formally recogmzes the 
collaborative work of faculty in the PDSN as well. This work includes university 
coordinator functions, school-based inquiry projects, participation on school-based 
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committees, work with PDS faculty on particular professional development activities, and 
instruction with candidates during the five-week lab experiences each semester. 

The university coordinator evaluation is used for faculty with formal liaison 
responsibilities in the PDSN. This evaluation instrument has a parallel version for PDS- 
based building coordinators. Items on the instrument reflect the jointly determined 
expectations for these roles. Educator preparation candidates and lab teachers, as well as 
role counterparts complete this evaluation instrument each semester. They are collected 
and compiled by the department chair. For university-based faculty the data from this 
evaluation are used in the annual performance evaluation. A procedure for formal and 
systematic review of the building coordinators based on data generated from this 
instrument has not yet been developed. Individual PDS building coordinators and their 
university coordinator covmterpart have informally reviewed the data and used it in 
refining their work in this role. Similarly, the department chair reviews the data for 
indicators of problems and concerns that might need to be addressed. This dimension of 
the performance assessment clearly requires further development. 

The Master Teacher Evaluation examines the master teacher’s performance along 
three dimensions: modeling research-informed best practices; ability to mentor, support 
and evaluate the educator preparation candidate’s performance; on-going active 
engagement in professional development. The master teacher evaluation process was 
initiated by the first cohort of master teachers in fall 1 998. It has been tailored and 
refined within each program by master teachers in conjunction wdth university faculty 
(see Thompson 2001). It serves as the basis for teachers maintaining their master teacher 
designation. 
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Currently, the data set related to P-12 student performance in the assessment 
framework is limited to the annual “State Report Card”, and then only the section 
reporting criterion-referenced test data. The CRCT is an evaluation linked to the state s 
Quality Core Curriculmn, the document which outlines achievement targets by subject 
area and grade level and around which teachers/schools are required to build their 
instructional programs (see PDSN Profile, Appendix A). The state requires and 
administers a commercially available norm-referenced test to all K-8 students. By their 
nature these are not tests that should be used to evaluate teacher performance or 
effectiveness. Furthermore, during the last two academic years 1999-2000 and 2000- 
2001, neither the State Department of Education nor the test vendor could/would validate 
the test results. Classroom specific dociunentation of instructional effectiveness 
constitutes the main source of data on student performance. For education preparation 
candidates the Impacting Student Learning components that they complete each semester 
constitute their primary evidence of influencing student achievement. 

Process Assessment 

Developing comprehensive and coherent collaborative procedures through which 
university and school-based faculty would equitably participate in educator preparation 
has been as much a goal of this initiative as has been improvements in program content 
and candidate performance. Monitoring and gauging the extent to which processes and 
procedures are effectively addressing all three goals are the focus of the process category 
of the assessment system. With regard to the role enactment evidence sources (master 
teacher, university and building coordinators) the process assessment focus relates to how 
effectively these roles serve to link the university and school-based components of the 
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programs; how well do these roles facilitate substantive communication, problem-solving 
and collaborative action with regard to both program implementation and candidate 
performance. The perceptual and satisfaction survey data sources provide indirect 
indicators of process effectiveness. In the first two semesters of PDSN implementation, 
for example, consistent negative responses by both candidates and lab teachers to an item 
regarding whether or not everyone was adequately informed about course requirements, 
prompted a review of these procedures by building and umversity coordinators. The 
review eventuated in procedural changes that subsequent survey data indicate everyone 
finds more satisfactory. 

Evidence sources that relate directly to PDSN operating procedures include the 
PDS Perception Survey, the Master Teacher Selection, PDSNI meeting notes, the 
comprehensive PDSN self-evaluation completed in spring 2000, the PDS Inquiry, 

Review and Renewal documents as well as a number of satisfaction surveys. As these 
collaboratively developed procedures are enacted, meetings and informal 
communications afford participants opportunities to endorse continuation, identify 
difficulties and challenges, and suggest changes. Maintenance of meeting and 
communication records provides an evidence base for reviewing and revising procedures 
as the PDSN evolves. One example in this area relates to the equitable representation of 
stakeholders on the university’s primary policy body for educator preparation programs. 
For two years after the PDSN began functioning the Teacher Education Coimcil included 
only one teacher and an administrator fi“om the PDSs. PDS university coordinators and 
building coordinators formulated a proposal that the Coimcil should include 
teacher/administrator representation from elementary, middle, and secondary levels. The 



Council and university adopted the proposal in Spring 2002, resulting in increased P-12 
representation including six representatives from three grade level groupings 
(Elementary, Middle, Secondary), and a practitioner in special education. 

At the end of the second full year of implementation, spring of 2000, each of the 
28 PDSs undertook a self-evaluation using the NCATE Draft Standards to organize their 
documentation of activities, strategies and practices. Once compiled by the PDSNI 
evaluator these data served as the basis for an overall self-evaluation. This self-study 
allowed us to gauge progress in the initiative against a relatively constant set of 
benchmarks. It also accommodated reviewing our work in relation to a broader national 
perspective on PDS work. Even at this early stage of implementation, this self-evaluation 
revealed considerable strength in the functional area of collaborative educator 
preparation. The preponderance of positive evidence related to this PDS function and its 
related standards. This was expected, as educator preparation was the primary focus of 
our collaborative work during this period. The professional development function 
emerged as the second strongest functional area in the self-evaluation. As faculty in the 
PDSN began working more closely with university faculty and educator preparation 
candidates they began to identify a professional development agenda for themselves. The 
PDSN sponsored workshops, seminars, and meetings to address this agenda. PDSN 
faculty also began to connect other school system-based professional development 
support to their PDS work. One of the dominant themes of the self-evaluation was the 
increased integration of educator preparation into the mainstream professional culture of 
each PDS. 
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While the PDS faculties acknowledged in the self-evaluation that their students’ 
academic achievement is their priority and they could point to considerable evidence of 
effort in this regard, they noted that they had not directly connected it to their PDS roles. 
Many noted that completing this self-evaluation had brought the connections between 
this function and the educator preparation function into focus. The self-evaluation also 
revealed a consistent dearth of evidence related to the inquiry function of the PDS. Each 
PDS also provided evidence of work on each of the then five standards in the NCATE 
PDS Draft Standards framework. These data were compiled to create a PDSN status 
snapshot. Three individuals reviewed the evidence set separately and generated a 
composite rating for each standard. They then reviewed their ratings together and came 
to consensus. The rating summary is provided in Appendix D. A narrative explaining 
the basis for each rating was generated and shared with the PDSN. From Network 
members’ collective review a follow-up agenda for the PDSN was formulated. A key 
element of that agenda was the creation of a formal four-year cycle of participation in the 
PDSN, which would include an Inquiry Year, and a Review and Renewal Year. The 
Inquiry Year would eventuate in each PDS pursuing a formal exploration of a topic or 
issue of interest to it, engaging faculty, administrators, university colleagues and educator 
preparation candidates in the process. It would formally encourage the inquiry function 
of PDS and situate it within the work and concerns of each PDS faculty. The Review and 
Renewal Year requires that each PDS continuously maintain evidence of its work in 
relation to the NCATE PDS Standards and that it formally review that evidence every 
four years to evaluate itself as a PDS. Based on its review, the PDS would determine if it 
wanted to continue in the PDSN and identify a self-improvement agenda for the next 
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four--year cycle. During Spring 2002 the PDSN generated a cycle structure that would 
begin in fall 2002 with approximately one-quarter of the PDSs situated in each of the fom 
years of the cycle. 

Through process assessment deliberate reflective attention is paid to the effects 
procedmes and processes are having on candidate performance and program 
implementation. Process assessment also can illuminate the extent to which procedmes 
and activities are supporting or inhibiting genuine collaboration among P-12 and higher 
education participants in the PDSN. 

Program Assessment 

The program category of the assessment system focuses on the outcome 
standards, understandings, skills, dispositions and results that candidates are expected to 
achieve and that procedmes and processes are intended to cultivate. Program reviews by 
external evaluators (e.g. national professional societies, the state certification agency, and 
the Regents’ Principles and Guidelines for Educator Preparation) constitute a major 
evidence somce for program assessment. These reviews attend to evidence in comse 
syllabi and sequences, of program coherence, depth, and comprehensiveness, alignment 
of program content with relevant scholarship, and, increasingly, evidence of program 
impact on candidate performances in relation to P-12 student achievement. 

Candidate performance evidence somces constitute a primary basis for program 
assessment, linking by inference candidates’ comse and lab performance evaluations to 
program quality and/or effect. Review of course performance assessment data and 
candidates’ portfolios figure prominently here. Candidates are expected to include in the 
portfolio actual performance evidence (an essay, a lesson or unit plan, a presentation 
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outline, a critical reflection on a lesson taught in lab) from graded course assignments, 
which the course instructor has keyed to one or more of the flSlTASC/Conceptual 
Framework Principles. A major challenge related to this component of program 
assessment is moving to a more direct examination of the course tasks and activities that 
form the basis for the candidates’ performance evaluations in relation to the 
INTASC/CFP they are purported in the course syllabus to address. Such aii exercise 
might exemplify the departure from tradition Stiggins (2000, pg. 20-21) asserts is 
embedded in authentic program assessment. 

The various perception and satisfaction surveys provide a continuous stream of 
evidence regarding participants’ views of program objectives, elements, and effects. The 
PDS Perception Survey includes a number of items related to integration of the class and 
field experience components of courses and the programs overall. The data set, which 
now includes eight successive administrations of the survey, indicates consistently strong 
positive perceptions of this integration across all participant groups. These items have 
been monitored closely, particularly in semesters when program refinements have been 
initiated. The employer satisfaction survey and the graduate satisfaction survey include 
items correlated to program adopted INTASC/Conceptual Framework Principles. The 
surveys attempt to gauge program effects by exploring respondents’ assessment of 
graduates’ performance effectiveness in relation to these principles. Data from the pilot 
survey of employers suggest above average satisfaction with graduates’ performance on 
items related to all ten flSlTASC/Conceptual Framework Principles. The strongest scores 
were on content knowledge and instruction (INTASC/CFP 1 «&7), and in meeting 
students’ developmental needs (INTASC/CFP 2). Scores did not vary significantly 



across grade levels or school type: PDS and non-PDS. Over time the data stream from 
these surveys should inform the assessment of program impact on the professional 
practices of graduates. The assessment of graduates’ effectiveness vis a vis program 
preparation is constrained by the extent to which the professional culture of the schools in 
which they are employed supports and nurtures practices that are grounded in that 
preparation. Additional strategies for investigating this alignment need to be developed. 

Referrals (or the lack thereof) under the terms of the guarantee also can be used as 
evidence of program effectiveness. None-the-less, more direct methods of assessing 
program impact on the teaching practices of graduates need to be developed, 
implemented, and integrated into the overall assessment system. 

Analysis 

The assessment system described in the preceding section of this paper is 
evolving in concert with on-going refinements in the educator preparation programs, 
cultivation of genuine university-school collaboration, and enhancements of candidate 
proficiencies in supporting student learning. Each of these categories is inherently 
complex. Their simultaneous pursuit can be overwhelming. Experience in this initiative 
suggests a dynamic complementarity and mutually supporting momentum between and 
across the categories with the assessment system serving as bridge and bond. The 
assessment system continues to be emergent in nature, prompted by questions, issues, 
concerns, and opportunities that themselves emerge in program enactment and 
collaborative activities. In this regard the assessment system is as much an opportunity 
for inquiry and professional development as for evaluation. It serves to both inform and 
transform practice, particularly with regard to educator preparation. But its utility in 



transforming P-12 teaching practices, expanding teachers’ participation in decision 
making and enriching their professional development agenda is becoming increasingly 
evident. 

Assessment as Evaluation and as Inquiry 

In a traditional vein assessment is generally thought of as distinct and deliberate 
activities or events imdertaken to provide evidence of accomplishment or achievement 
and a basis for a qualitative evaluation of that evidence. In recent years considerable 
professional attention has been paid to the form that assessment activities take, 
particularly with respect to the extent to which the assessment activity aligns both with 
the type of learning it is expected to evidence and with as authentic a demonstration of 
the learning in action as possible. In educator preparation this concern is evident in 
expectations (both internal and external) that candidates demonstrate proficiency in 
instruction related tasks (e.g. instructional planning, execution, evaluation, revision; 
classroom management; effective work with students of diverse backgroimds; collegial 
professional traits) and in affecting their students’ learning and achievement. 

Within the system described in this paper, candidates’ performances in relation to 
program standards (INTASC) are embedded in courses/labs that comprise the program 
curriculum. Candidate performance evidence is summarized in course grades, and is 
compiled in considerable detail in their portfolios. Similarly the Impacting Student 
Learning (ISL) components, which are strategically situated within designated courses in 
each program, provide proficiency evidence that contributes to course grade 
determinations and the candidate’s portfolio. Different evaluative questions will prompt 
examination of different arrays of candidate performance evidence. The electronic data 
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base created to compile and store individual candidate’s INTASC related scores, lab 
performance, professional qualities, and ISL scores by course can be aggregated in varied 
ways to represent class and cohort proficiencies, patterns related to the INTASC 
standards, and by extension program quality. Individual portfolios provide even more 
detailed and elaborated evidence of proficiencies, specific patterns of strengths and 
weaknesses, areas of particular growth and development within the program, and areas of 
stability — seemingly imaffected by program specific interventions. As Scannell (2000) 
notes evidence compiled in assessment systems will be of interest to different audiences 
and at varying levels of detail. The capacity to manipulate compiled evidence in varied 
ways, in relation to varied factors, and at different levels of aggregation enhances the 
utility of the assessment system to inform performance and program evaluation. 

With regard to program evaluation the assessment system accommodates 
inferences from candidate performance evidence, program enrollment patterns, and 
graduates’ employment rates to program quality. The more direct evidence sources 
include the various perception and satisfaction surveys as well as the periodic formal 
program reviews by external approval and accrediting agencies. By administering varied 
perceptual surveys during every semester of PDS implementation a continual data stream 
is created through which effects of program and personnel changes as well as the impact 
of environmental forces can be detected and tracked. The PDSN Perception Survey has 
been quite sensitive, for example, to changes in participating umversity faculty, 
perceptions of candidates’ preparedness relative to their cohort status (first, second, third 
semester), and external (particularly political) environmental forces affecting P-12 



education. 



The following examples illuminate this point. With regard to personnel changes, 
secondary PDS faculty survey data became significantly more positive when both 
university coordinators changed. The replacement faculty made a concerted effort to 
strengthen communications with and among the secondary PDS building coordinators 
and to ensure coordination of placements, seminars, and meetings across all five 
secondary PDSs. From the first administration of the survey PDS teachers responses 
demonstrated patterns of difference in their assessments of candidates’ preparedness 
depending on candidates’ cohort (longevity in the program) standing. With some 
refinements in the survey instrument we have been able to track this pattern of 
differentiation and have found it to be consistent across teachers’ and candidates’ survey 
responses. In the spring of 2000 the new governor proposed a sweeping education 
reform agenda, promoting it in part by blaming teachers for the state’s poor scoring 
(relative to other states) on norm-referenced tests, and excluding them from participating 
in building the reform agenda. 

During the semester when this was happening the PDS Perceptual Survey ratings 
on items related to school climate and teachers’ willingness to work with candidates 
dropped significantly. Investigating the negative survey results revealed considerable 
anxiety on the part of teachers related to the governor’s reforms, not dissatisfaction with 
the PDS program. Candidates’ ratings of the same items declined also. This was due to 
their sensing the teachers’ anxiety and misunderstanding it as lack of interest and 
enthusiasm for having the candidates in their schools and classrooms. This episode 
provided an opportunity for bolstering inter-institutional support and collaboration. It 
also highlighted the importance of monitoring data and investigating the sources 
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influencing perceptions that are reflected in the data prior to drawing conclusions from 
the data and planning action based on those conclusions. 

Assessment as inquiry emphasizes the integral link between processes and 
outcomes. Within the context of the assessment system this view of assessment 
encourages critical examination of processes and deliberate actions undertaken in pursuit 
of collective goals, both within and across the categories of performance, processes, and 
progreim. Performance scores such as those provided in the appendices of this paper 
smnmarize candidates’ improvement from mid-term to final evaluation in the 
Apprenticeship, or smnmarize evidence of P-12 student progress and achievement as 
recorded in sample ISLs. These score smnmaries reveal nothing about what processes, 
ways of thinking about and using knowledge informed candidates’ progress and learning. 
When c^didates’ entire ISL reports are reviewed, evidence of their thinking, their 
attention to students’ agency in learning, and their own questions and concerns as they 
worked through the instructional unit are illuminated. These reports document the 
working knowledge candidates are employing in their instructional work. They 
illuminate both the distinct personal sense-making of individual candidates as well as 
shared understandings that are being enacted. Assessment as inquiry requires this 
looking within learning performances and behind summary scores in order to 
comprehend and appreciate more fully the dimensions and quality of the learning 
outcome as evidence of learner agency. Candidates are encouraged by faculty to review 
these documents as a basis for examining their learning, how they are comprehending and 
applying it, and how it is shaping their professional identities. Within the “Teaching for 
Understanding” fimnework this strategy reflects the on-going and recursive qualities of 
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assessment in the learning process (Wiske 1998). It begins to approximate 
Delandshere’s (2002) inquiry metaphor for assessment. 

Assessment as inquiry serves a similar role in both the program and process 
categories of the assessment system. In the process category, for example, the NCATE 
PDS Standards serve as a framework for undertaking, recording and reviewing the 
quality of our collaborative processes. The narratives generated by each PDS in the 2000 
self-evaluation are revealing of how faculty within each PDS comprehended and 
prioritized the standards, and how they categorized their work in relation to them. The 
narratives reveal commonalities as well as distinctions across the PDSs regarding their 
perceptions of the emerging collaboration with the university. These in turn provide a 
basis for systematic inquiry into how institutional climate and its professional culture 
influence its participation in an inter-institutional partnership. 

For the educator preparation unit, conceptualizing assessment as inquiry as well 
as evaluation affords an opportunity for defining and pursuing with P-12 partners an 
extensive applied research agenda. The PDSN environment can distinctively support 
examining contextual and cultural influences on and interactions with performance 
development and quality, on inter-institutional collaborations, and on enactments of 
educator preparation program elements. All of these are as yet inadequately understood 
dimensions of educator preparation and professional practice; and often left unexamined 
in traditional research. 

Assessment Informing and Transforming Practice 

As Stiggins, Weisenbach, and Stroble note the development, implementation and 
full utilization of a comprehensive assessment system are likely to challenge well- 
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entrenched traditions. Stiggins suggests that deliberate, concerted attention to assessment 
of candidate performances would itself be a major innovation in educator preparation, 
and that the introduction of an assessment system would signal a transformational 
threshold in higher education. Weisenbach and Stroble both suggest that by including all 
key stakeholders in developing the assessment system individuals relinquish exclusive 
and private prerogatives in defining course expectations, teaching practices, standards 
and performance requirements. In this work for example, in order to convey to candidates 
that field experiences were an integral component of program course-work university 
faculty determined that field experience performances had to carry substantial weight in 
the calculation of course grades. This determination necessitated more substantive 
collaboration between university and PDS-based faculty regarding program/course 
content, candidate performance expectations, and opportunities for candidates to 
undertake required tasks in the lab settings. As this work progressed the PDS-based 
faculty identified a professional development agenda through which they could enrich 
their own understandings and skills for this more intensive responsibility with educator 
preparation candidates. This self-assessment much like the individual PDS self- 
evaluations in 2000 revealed much about the views of the PDS partners regarding both 
the collaborative partnership and program enactment. Over time assessment has 
increasingly become an integrated component of the unit’s work, informing, challenging, 
and bridging performance, process, and program focused endeavors. Most significantly, 
assessment has eventuated in greater openness and more substantive engagement of 
participants in the educator preparation partnership. This has created opportunities for 
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PDSN faculty to assume leadership roles, and to examine more deeply issues of practice 
with university colleagues. 

Challenges 

Developing and sustaining a comprehensive assessment system requires 
co mmitm ent of resources — human, time, financial. These require an institutional and 
unit commitment to this endeavor as integral to the distinct mission of educator 
preparation. External accreditation and program approval standards can partially 
leverage this work, but they will not serve as a stable foundation. Commitment to and 
advocacy for such a system by participating university and school partners alike are 
pivotal. Developing the assessment system described in this paper was accomplished 
primarily through reallocation of existing resources and strategic plaiming, and revised 
procedures. The system continues to be challenged by less than adequate resources that 
would support more timely data compilation, dissemination and analysis to inform 
refinements and revisions across the program, process, and performance categories. 
Creative staff redeployment might well address this challenge. In periods of reduced 
resources competing priorities can constrict fully implementing and sustaining a 
comprehensive assessment system. Creative stewardship accompanying unwavering 
commitment are required to stay the course during such periods. 

The culture of higher education does not readily embrace systematic assessment, 
traditional recognition and reward systems are not well structured to acknowledge the 
energy and attention such systems demand of faculty, and faculty seldom come to the 
enterprise well prepared or positively disposed to it. To the extent that developing and 




Page 29 

30 



sustaining a comprehensive unit assessment system is pursued with integrity it will face 
cultural, resource, and competing priority challenges. 

Forces in the political arena pose the most daunting challenges to systematic 
assessment, as well as essential revisions in educator preparation as represented in the 
NCATE 2000 Unit Standards. The program described in this paper resides in a public 
university that is governed by a vmiversity system Board of Regents. It is Regents’ policy 
that public vmiversity educator preparation programs must be NCATE accredited. Private 
colleges and vmiversities do not come imder the piuwiew of the Board of Regents, and the 
state certification agency which also approves educator preparation programs does not 
require all programs to meet NCATE Standards. Over the last two years this same 
agency has itself become a purveyor of an alternative route to certification. In practice 
thi s three tiered system disadvantages the public vmiversities. The standards they must 
meet require considerable resource investment and may more rigorously limit access to 
their programs. Standards then, particularly in periods of teacher shortages, become in 
the parlance of the political arena “uimecessary barriers” to “promising individuals” who 
might otherwise be interested in pursuing careers in education. Simultaneously, public 
vmiversities are expected to increase their capacity for candidates, and their funding is at 
least partially based on FTE. The lack of consistent standards and expectations for all 
educator preparation programs in a state can vmdermine those held to more rigorous, 
evidence-based standards. 

A second challenge to comprehensive assessment systems posed within the 
political arena is the current inclination to utilize a single measure to evaluate candidate 
performance and program quality. Recently adopted federal regulations exemplify this 
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phenomenon. In this environment, there is a certain economy if not wisdom (albeit 
cynical) in investing limited resources in test preparation activities and foregoing multi- 
faceted, comprehensive assessment systems. These challenges do not bode well for 
improvements in educator preparation, candidate proficiency, P-12 student learning and 
achievement. Left imchallenged these forces will completely undermine the goals of the 
NCATE 2000 Unit Standards, and the work of NCATE members and affiliates. 

The NCATE 2000 Unit Standards, themselves, challenge educator preparation 
units. To enrich their curriculum by fully integrating theory and practice, to collaborate 
substantively with P-12 colleagues, and, through the assessment system standard, to 
provide extensive evidence of the quality of their graduates and the positive effects their 
programs have on education practitioners and P-12 student learning and achievement. 

The assessment system described in this paper illustrates one unit’s efforts to meet this 
challenge. Through this assessment system data from multiple sources affirm and 
substantiate evaluative judgments regarding candidates’ performances, program quality, 
and process effectiveness. The assessment system facilitates critical review of the unit’s 
work in relation to clearly articulated standards and in light of performance-based 
evidence. It is complex, demanding of time resources and most importantly commitment. 
It holds considerable promise for contributing to the quality of education practitioners 
and ultimately to the enrichment of P-12 student learning and achievement. 
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APPENDIX A 

Professional Development School Network Profile 

2002 



Suburban: 

Elementary: 5 
Middle: 2 
Secondary: 2 

Secondary: 0 



PROFESSIONAL DEVELOPMENT SCHOOL NETWORK PROFILE-2002 
PDS Demographic Diversity: 

Urban: Rural: 

Elementary: 6 Elementary: 5 

Middle: 2 Middle: 1 

Secondary: 2 Secondary: 1 

Title I: 

Elementary: 7 Middle: 2 

Fully half of all the PDSs (13) have student populations in which 40% or more students 
are eligible for free or reduced lunch. 

Ethnicity: Majority Minority: 13 Majority White: 13 

In only toee of the schools the “minority” population is less than 20% of the entire 

student population. 

Elementary PDS Achievement Indicators: *3 PDSs do not have grades above 

4“^ Grade CRCf* 

# 70% passing 

# improving from 2000 

# at or above state average 

S**’ Grade Writing* 

# 80% at or above 11/13 

Stage 4 (6 Stage Scale) 

State average was 80% 

# improving from 2000 7/13 

Middle PDS Achievement Indicators: 



Reading 


Lang.Arts 


Math 


11/13 


12/13 


7/13 


10/13 


9/13 


4/13 


11/13 


11/13 


10/13 



6"^ Grade CRCT 


Reading 


Lang. Arts 


Math 


# 70% passing 


5/5 


3/5 


3/5 


# improving from 2000 


4/5 


4/5 


4/5 


# at or above state average 
8*^ Grade CRCT 


3/5 


3/5 


3/5 


# 70% passing 


5/5 


4/5 


2/5 


# improving from 2000 


5/5 


4/5 


4/5 


# at or above state average 


4/5 


4/5 


4/5 



8**^ Grade Writing 



# 80% at or above “On Target” 

# average above state 

# improving from 2000 

Secondary PDS Achievement Indicators: 



5/5 

4/5 

5/5 



GHSGT 


Eng. 


Math 


SS 


Sci 


AllComp. 


Wrtg. 


# 90% 


5/5 


4/5 


2/5 


0/5 


0/5 


2/5 


passing on 
admin. 














# 80% 


5/5 


5/5 


3/5 


2/5 


2/5 


5/5 


passing on 
1®* admin. 

# improving 


3/5 


2/5 


2/5 


3/5 


3/5 


3/5 


from 2000 
# at or above 


4/5 


3/5 


3/5 


2/5 


2/5 


2/5 



State average 
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APPENDIX B 

Impacting Student Learning Components, Schedule, Rubric 
Sample Data, Spring 2002 
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Impacting Student Learning— Elements of the ISL Component 



Student Profiles: 

Developmental characteristics 
Background and experience 
School, classroom, community contexts 
Learning styles, abilities, needs 
Interviews of students, teachers, others 



Standards for Students: Learning: 

(QCCs, National Content Standards, 
Stanford 9) 

Content Understanding 

Type of learning task: knowledge, concept, 

skill, application, theory, disposition 

Thinking processes 

Outcomes, goals and objectives 



Standards for Teachers: Pedagogy: 

(Conceptual Framework/National Board) 
Developmentally appropriate 
Multiple paths to leaming/diversity 
Cognitively/actively engaging 
Performance-based assessments 



Assessment of Student Learning: 

Tools 
Processes 
Pre & post 

Formative & summative 
Authentic 

Connections to standards 



Evaluation of Student Learning: 

Artifacts 

Analysis 

Explanation 



Reflection on & Refinement of Teaching 
& Learning: 

Continual reflection & refinement during 
teaching 

Final reflection & analysis 
Both Include: 

Implications for further learning 
Refinements/revisions needed 
Action plan/next steps 
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IMPACTING STUDENT LEARNING (ISL) COMPONENTS BY PROGRAM BY LEVEL OF FOCUS 
All ISLs are embedded in Content Pedagogy Courses or the culminating Apprenticeship, Semester IV. 
Field Experiences comprise five dedicated weeks of each fifteen-week semester. 



Certification 

Program 


Individual/ 
^u to rial 


Whole Class 
Instruction 
4-7 Student Focus 


Whole Class 
Instruction 
Whole Class Focus 


Varied Levels of 
Focus 

Multiple Examples 


Early Childhood 
Program (PK-5) 


Semester I: Math 
content focus, 1-3 
students, 3 weeks 


Semester II: Science 
content focus, 4-7 
students’ 
performance 
assessed, 2-4 week 
unit 


Semester III: Social 
Studies, whole class, 
3-4 week unit; & 
Reading diagnosis- 
remediation, 2-3 
students, 4 weeks 


Semester IV: 
Candidate 
completes multiple 
units across varied 
subject areas, fully 
integrated into 
classroom 
curriculum, jointly 
determined with 
master teacher 


Middle Grades 
Program (4-8) 


Semester I: Content 
concentration focus 
(each candidate 
pursues 2 content 
concentrations: 
Math, Science, 
Social Studies, 
Language Arts), 
assessment focus is 
on learning 
processes, four 
weeks 


Semester II: Content 
concentration focus, 
whole class 
performance 
assessed and 
Reading diagnosis- 
remediation tutoring 
of three students 
four weeks 


Semester III: 
Interdisciplinary 
focus, whole class 
performance assessed, 
five weeks 


Semester IV: 
Candidate 
completes multiple 
units across content 
concentration areas, 
fully integrated into 
classroom 
curriculum, jointly 
determined with 
master teacher 


Secondary 
Programs in single 
subject majors (7- 
12) 


Semester I: 
Precedes formal 
entry to program 


Semester II: 
Certification content 
field, 4-7 students’ 
performance 
assessed across 
multiple classes, 3-4 
weeks 


Semester III: 
Certification content 
field, whole class 
performance assessed, 
five weeks 


Semester IV: 
Candidate 
completes multiple 
units within 
certification content 
field, fully 
integrated into 
classroom 
curriculum, jointly 
determined with 
master teacher 
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PROFESSIONAL DEVELOPMENT SCHOOL NETWORK 
Impacting Student Learning 
Scoring Rubric Criteria 

4 = exemplary performance on all components of ISL task — goals are well-stated and 
appropriate for grade level/subject and students; learning profiles are well-developed, complete, 
and accurate; diagnosis of student needs is based on data; instructional practices and strategies 
are active, student-centered, and appropriate to diagnosed needs and the task; assessments are 
appropriate for the type of learning being assessed; reflections reveal reasoned and supported 
analysis of lessons, student achievement, and future needs; there is explicit evidence of 
understanding of students as learners; writing and/or speaking are clear, strong, and thoughtful; 
references and examples are integrated into the analysis; consistently correct usage of standard 
English, consistent and accurate use of APA style. 

3 = proficient performance on all components of ISL task — goals are appropriate for grade- 
level; learning profiles give an adequate but not complete insights; diagnosis of student needs is 
based on accurate but insufficient data; instructional practices and strategies are subject-centered 
and appropriate to the task; assessments address the learning task(s); reflections reveal well- 
supported analysis of lessons and student achievement; there is some evidence of understanding 
of students as learners; writing and/or speaking are clear and coherent; some evidence of 
integration of references and examples; consistently correct usage of standard English, consistent 
and accurate use of APA style. 

2 = efforts to address all components of ISL task are in progress; performance on all 
dimensions of task is not completely accurate or coherent — goals are related to content; learning 
profiles are insufficient for accurate planning; planning is superficial and lacks thoughtfulness; 
instructional practices and strategies are subject- or teacher-centered and don’t fit well with 
objectives; reflections and analysis are descriptive and superficial rather than reasoned and 
supported; references are dropped-in rather than integrated; inconsistencies in correct usage of 
standard English, inconsistent and inaccurate use of APA style. 

1= performance is unsatisfactory; not all areas are addressed; performance is inaccurate, 
incoherent, lacks clarity - goals are incomplete or poorly stated or inappropriate; profiles are 
incomplete and/or inaccurate; planning is minimal; instructional practices and strategies are not 
thoughtful or well planned, inappropriate to content focus and/or learners; reflections and 
analysis are not at all evident; faulty reasoning; consistently incorrect usage of standard English, 
consistently inaccurate use of APA style. 
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IMPACTING STUDENT LEARNING ASSESSMENT RUBRIC 



Rating Scale: 

4 — Exemplary level 

3 - Proficient level CFP - CONCEPTUAL FRAMEWORK 

2 - In progress level PRINCIPLES ADAPTED FROM INTASC 

1 - Unsatisfactory level 

Student: Program: Block: Course: 

1. Learning Goals: (CFP-1) 

• Designates goals in terms of knowledge, skills, theory concepts, and dispositions. 

• Describes the goals and how they represent the above categories. 

• Embeds content. 

2. Learning Profiles: (CFP - 2, 3) 

• Includes profile of each student as a learner (developmental characteristics, strengths, weaknesses, learning styles, 
outside variables, experience, prior knowledge). 

• Describe implication of the learner profiles for teaching. 

• Describe learning context and its implications for teaching. 

• Provides evidence in support of profiles. 

3. Planning for learning: 

• Describes diagnostic assessment: 

o Instrument/Performances 
o Student data 
o Implications for teaching 

• Uses effective lesson planning: 

o Objective (s) 
o Assessment 
o Procedures 

• Evidences appropriate teaching practices 

o Variety 

o Critical creative thinking, problem-solving 
o Appropriate communication and technology 
o Support learning of all students 
o Positive social interaction 
o Active student engagement 

4. Student Achievement: (CFP — 8) 

• Describes formative assessments: 

o Instruments/Performances 
o Student achievement data 
o Describes needed student intervention areas 
o Implications for teaching 

• Describes summative assessment: 

o Instruments/Performances 
o Student achievement data 
o Analyzes the ‘Vhy” related to achievement 
o Implications for teaching 

5. Reflection and Refinement: (CFP - 9) 

• Reflects upon own instruction and the impact on student achievement 

• Indicates next steps for instruction with each learner profiled. 

Overall Score 



(CFP-4, 5, 
6, 7,8) 




Page 41 

42 



BEST COPY AVAILABLE 



Preliminary ISL Evidence of Teacher Candidate Ability to Support P-12 Students’ Achievement 

Fall 2001: ECED 4313 — Reading Diagnosis and Remediation 

Summary data on tutorial outcomes for 49 elementary students in PDSs, grades K-5, tutored by teacher 
candidates. Data are based on student pre-tutoring and post-tutoring scores on the Informal Reading Inventory 
(IRI). 

Change for Entire Sample Change for Improved Group 



Indeoendent 


1.37* 




2.4* 


Reading Level 
Instructed 


1.55* 




2.05* 


Reading Level 

* Average Reading Level Increase 


%Increase 


%Same 


%Decrease 


Independent 


57% 


37% 


6% 


Reading Level 
Instructed 


76% 


18% 


4% 


Reading Level 









Percents represent percent of 49 students in each category of change from pre to post-tutoring assessment. Tutoring 
period was @4 weeks, approximately two hours a week per student. 

Spring 2002: MGED 3222— Integrated Reading to Learn 

Summary data on tutorial outcomes for 1 5 middle level students in PDSs, grades 6-8, tutored by teacher 
candidates. Data are based on student pre-tutoring and post-tutoring scores on the Informal Reading Inventory 
(IRI). 





Pre-tutoring Average 


Post-tutoring Average 


#Improving 


Word Recognition 


4.8 


6.1 


10 


Independent Level 
Word Recognition 


6.4 


6.8 


6 


Instructional Level 
Comprehension 


5.4 


6.5 


9 


Independent Level 
Comprehension 


6.1 


6.8 


7 



Instructional Level 

The IRI does not reflect scores above the 8*** grade level; a number of students scored at that level on the pre-tutorial 
assessment, so there are not post-tutoring scores available for them. Tutoring took place during the 4--week 
intensive lab period of spring 2002; in a number of instances tutoring was precluded for SAT-9 test preparation. 

Spring 2002: MGED 3241 — ^Mathematics Pedagogy 

Summary data were only available from two students’ Impacting Student Learning reports. In one the 
average score on the pre-assessment was 37.42; while the post-assessment score average was 87.98. In the other 
ISL, of the 16 students in the class, 12 realized increased post-test scores, two remained the same, two declined. Of 
the 12 whose scores increased, 9 increased by 50% or more. 

Spring 2002: SCED 4401 — Secondary Science Pedagogy I 

One candidate’s ISL included pre and post assessment error rates on two pairs of comparison classes: 



Pair I: Average # missed 13.8 8.33 

Average # correct 1 1 .635 1 6.66 

Pair II: Average # missed 12.364 6.394 

Average # correct 12.636 18.606 
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Spring 2002: ECED 3231 — Science Pedagogy I 

In this course teacher candidates taught a whole class unit, and monitored the pre and post instructional 
assessments of five particular students for the Impacting Student Learning component. The units were fiilly 
integrated within the science curriculum of the class in which the teacher candidate was completing her/his lab 
assignment. The following are representative summaries from teacher candidates’ post-ISL documentation. 



Candidate I: 


Pre-assessment 


Post- Assessment 


Topic 1: 


4 = 100% correct 
1=2 errors 


5 = 100% correct 


Topic 2: 


4 = 100% correct 
1 = 1 error 


5 = 100% correct 


Topic 3: 


0 = 100% correct 

2 = 3 errors; 2 = 2 errors 

1 = 1 error 


5 = 100% correct 


Candidate 11: 


Student 1 : 


1 correct 


5 correct 


2: 


1 correct 


5 correct 


3: 


3.5 correct 


5 correct 


4: 


4 correct 


5 correct 


Candidate III: 


Topic 1 : 


10% correct 


80% correct 


Topic 2: 


20% correct 


80% correct 


Topic 3 : 60% correct 

(%s are class average of correct answers) 


100% correct 


Candidate IV: 


Student 1 : 


5 errors 


1 error 


Student 2: 


2 errors 


0 errors 


Student 3: 


4 errors 


1 error 


Student 4: 


0 errors 


4 errors 


Student 5: 


0 errors 


0 errors 


Candidate V: 


Student 1: 


15 


43 


Student 2: 


30 


50 


Student 3: 


10 


50 


Student 4: 


20 


37 


Student 5: 


10 


41 



(This assessment included ten items, each worth five points for a total of 50 possible points. Data represent 
points earned by each student on the pre-test and the post-test.) 



Spring 2002: MGED 4210 — ^Apprenticeship, Pre and Post assessments on six 
Impacting Student Learning Units completed by six Apprentices. 



Content focus 


Pre-assessment Mean 


Post-assessment Mean 


Grade 


Geometry 


17.6% 


89.6% 


7- 

7- 

8* 


Geometry 


34.5% 


83.8% 


Quadrilaterals & 
Polygons 


17% 


75% 


Invertebrates 


62.8% 


83.7% 


^ID 


Astronomy 


45.6% 


89.3% 


s -s 
00 00 


Literature 

(Diary of Ann Frank) 


28% 


83% 
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APPENDIX C 

Apprentice Mid-term and Final Evaluation Summaries 



Apprentice Mean Scores (1-4 scale) by 
Program on Conceptual Framework Principles 

Baseline data for 2001-2002 academic year. Sample of 20% or greater for each program 





midterm 


final 


Midterm 


final 


midterm 


final 


10 


3.44 


3.87 


3.28 


3.70 


3.88 


4 


7,9 


3.13 


3.74 


2.91 


3.63 


3.28 


3.66 


1,2, 3, 4,6 


3.24 


3.79 


2.64 


3.57 


3.25 


3.61 


5 


3.21 


3.76 


3.14 


3.57 


3.25 


3.72 


8 


3.03 


3.68 


2.71 


3.68 


3.35 


3.70 



Percentage of apprentices by program with score increase from midterm to flnal 
CF Principles ECED MGED SCED 



10 


90 


100 


50 


7.9 


90 


100 


100 


1,2, 3, 4, 5, 6 


90 


100 


100 


5 


90 


87.5 


100 


8 


90 


100 


50 



Percentage of apprentices by program with a score of proflcient (3) or above at final 
CF Principles ECED MGED SCED 



10 


100 


100 


100 


7,9 


100 


100 


100 


1,2, 3, 4, 6 


100 


100 


100 


5 


100 


100 


100 


8 1 


100 


100 


100 



Percentage of apprentices by program with an exemplary score (4) at flnal 
CF Principles ECED MGED SCED 



10 


60 


0 


100 


7,9 


50 


25 


25 


1,2, 3, 4, 6 


60 


12.5 


25 


5 


60 


12.5 


0 


8 ^ 


50 


37.5 


25 



Gains of apprentices from midterm to flnal by program 



CF Principles ECED MGED SCED 



10 


.43 


.42 


.12 


7,9 


.61 


.72 


.38 


1,2, 3, 4, 6 


.55 


.93 


.36 


5 


.55 


.43 


.47 


8 ^ 


.65 


.97 


.35 
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Professional Development School Network — Self-Study 2000 
Composite Summary Scores for PDS Functions and NCATE Draft Standards 
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PDSNI— Self— Evaluation using NCATE DRAFT PDS STANDARDS (1997) 

1998—2000 



Composite Scores (1-3 Sea 


e) for Four PDS Functions 


PDS Function 


1 Parallel 


2 Cooperation 


3 Collaboration 


Educator 

Preparation 






3 


Professional 

Development 




2.6 




Research & Inquiry 


1.6 






Support P-12 
Student Learning 




2.5 





Composite Scores(l-4 Scale) for Five Draft Standards 



STANDARD 


1 Beginning 


2 Developing 


3 Acceptable 


4 Target 


I. Learning 
Community (5) 




2.2 






II. 

Accountability 
& Quality 
Assurance (5) 




2.46 






m. 

Collaboration 

(3) 




2.8 






IV. Diversity (3) 




2.47 






V. Structures, 
Resources, and 
Roles (4) 




2.73 







( )=Number of discrete elements in Standard for which a score is calculated. 




49 

Page 47 



PDSNI— Self— Evaluation using NCATE DRAFT PDS STANDARDS (1997) 

1998—2000 



Composite Scores on Elements making up the 


NCATE DRAFT Standards 


Standard Elements 


1 Beginning 


2 Developing 


3 Acceptable 


4 Target 


I. Learning 
Community 










Support Learners 




2.7 






Inquiry-based Practice 


1.5 








Shared Professional 
Vision &Grounded 
Knowledge Base 




2.7 






Serve as Instrument of 
Change 




2 






Extended Learning 
Community 




2.2 






n. Accountability 
& Quality 
Assurance 










Develop Professional 
Accountability 




2.3 






Assure Public 
Accountability 


1.5 








Set PDS Participation 
Criteria 






3.2 




Develop Assessments, 
Collect Information and 
Use Results 






3 




Engage with the PDS 
Context 




2.3 






HI. Collaboration 










Engage in Joint Work 




2.7 






Design Roles & 
Structures to Enhance 
Collaboration & Develop 
Parity 




2.7 






Systematically Recognize 
& Celebrate Joint Work & 
Contributions of Each 






3 




IV. Diversity 










Ensure Equitable 
Opportunities to Learn 




2.6 






Evaluate Policies & 
Practices to Support 
Equitable Learning 
Outcomes 


1.5 








Recruit & Support 
Diverse Participants 






3.3 




V. Structures, 
Resources & Roles 










Establish Governance & 
Support Structures 




2.2 






Ensure Progress toward 
Goals 




2.7 






Create PDS Roles 






3 




Use Effective 
Communication 






3 





Composite scores are based on review of 26 of 28 PDS Self-studies, completed in spring 2000. 
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